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(57) Abstract: An image processing 
method obtains a glassless image from a 
color frontal facial image bearing glasses 
by using a recursive principal component 
analysis (PCA) reconstruction. The 
image processing method comprises 
the steps of a) receiving an RGB color 
frontal facial image; b) extracting 
candidates of eye regions from the 
received RGB color frontal facial image; 

c) determining an exact eye region out 
of the candidates and normalizing the 
received RGB color frontal facial image; 

d) extracting a glasses frame region by 
using color information contained in 
the received RGB color frontal image 
and edge information of a glasses 
frame; e) performing an RGB -HIS 
transformation on the normalized frontal 
facial image; f) generating H\ S\ and 
I' glassless compensated images; g) 
obtaining R\ . G\ and B' compensated 
images by performing an HIS-RGB 
transformation on the H\ S\ and V 
glassless compensated images; and h) 
creating a glassless final color facial 
image on the basis of the R\ G\ and B' 
compensated images. 
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IMAGE PROCESSING METHOD FOR REMOVING GLASSES 
FROM COLOR FACIAL IMAGES 

TECHNICAL FIELD 
5 The present invention relates to image processing fields, and more 

particularly, to an image processing method for removing glasses from a color facial 
image by using recursive principal component analysis (PCA) reconstruction. 

BACKGROUND ART 

10 As an information-oriented society has been come, person identification 

techniques for discriminating a person from others have become more important, and, 
thus there have been significant number of studies in the field of personal 
information protection and person identification through a computer using 
biometrical technologies. In biometrical technologies, facial recognition technique 

15 becomes the most convenient and competitive technique since it does not require a 
specified action or behavior of a user and employs a non-contact manner. The 
facial recognition technique is widely used in various applications such as 
identification, human-computer interface (HCI), and access control. However, 
there are several drawbacks in the facial recognition technique. One of these 

2 0 drawbacks is deformation of facial images occurred by glasses. 

To remove glasses from a facial image with the glasses, various image 
processing methods are proposed: an algorithm for extracting glasses from a facial 
image using a deformable contour to remove the extracted glasses; an algorithm for 
removing small occlusion regions such as certain facial regions occluded by glasses 

2 5 using a flexible model that is called as an active appearance model; and an image 

processing method using PCA algorithm. 

An image processing method using PCA algorithm is now widely used. 
The PCA algorithm is classified into two processes. One is a training process for 
extracting eigenfaces from a plurality of unspecified sample glassless facial images 

3 0 T N , wherein N=1,2,...,M. The sample facial images F N include facial images of 

an individual and/or another individuals. The other is a process for obtaining 
glassless reconstruction images from current input facial images T with glasses by 
using the extracted eigenfaces. 

Descriptions of the training process for extracting eigenfaces will be first 
3 5 described in detail. An average image <p are calculated from the sample facial 
images T N for use in the training process by using Equation 1 and the average 
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image <p is subtracted from the sample facial images T N as expressed in Equation 
2, wherein each of the sample facial images T N is expressed as a column vector. 

5 

® N =r N -<p (Eq.2). 

Then, a covariance matrix C with respect to the sample facial images is 
obtained from differential images ® „ , which is calculated by subtracting the 
1 0 average image cp from each of the sample facial images F N by using the 
following Equation 3. 



Mfc x n n (Eq.3) 



15 wherein A is a matrix composed of the differential images <3> N and A T is a 
transpose of A. 

Consequently, eigenvectors are obtained from the covariance matrix C, 
wherein the eigenvectors is referred to eigenfaces u k (k=l,--,M). Detailed 
description for a process of obtaining the eigenfaces u k will be omitted because this 
2 0 process is well known to those skilled in the art. 

Next, the input facial images T with glasses are expressed as glassless 

reconstruction images t by using the eigenfaces u k . With the following 

Equation 4, the average image <p is subtracted from the input facial images T , and 
the resultant is projected to the respective eigenfaces u k . 



25 



co k =ul(T-<p) 9 ^ ^ 



wherein ^ is a weight which allows the input facial images T to be expressed on 
a space consisting of the eigenfaces « & . The reconstruction images f are also 
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expressed in terms of the sum of weights of the eigenfaces u k extracted from the 
sample facial images T N by using the following Equation 5. 

f = P + f>'""' (Eq.5) 
M'<M 

5 

wherein a number of the eigenfaces u k required is equal to M or to M' less than M, 
M being a total number of the eigenfaces u k . 

Where eigenfaces u k are extracted from the sample facial images T N as 
described above, the extracted eigenfaces u k include facial characteristics only so 

1 0 that final glassless facial images can be obtained by reconstructing the input facial 
images r on the basis of the extracted eigenfaces u k to produce the 
reconstruction images T . However, the reconstruction images f produced 
according to the conventional method have many errors thereon. Referring to Fig. 1, 
which shows that glasses are not removed completely although the reconstruction 

15 images r are similar to the input facial images T , and there are numerous errors 
over the reconstruction images f . In Fig. 1, "client" is a person included in a 
training set and "non-client" is a person excluded in the training set. Although there 
are numerous errors as shown in Fig. 1, the reconstruction images f of "clients" are 
better than those of "non-clients" in quality since facial characteristics are reflected in 

2 0 the extracted eigenfaces u k . 

However, there are some problems in regarding the reconstruction images 
r obtained according to the conventional method as complete glassless facial 
images. Firstly, if the reconstruction images f are generated with respect to the 
input facial images F on the basis of the eigenfaces u k that are extracted from the 
25 sample facial images T N included in the training set, particular characteristics of the 

input facial images T would not be appeared on the reconstruction images f . 
Secondly, if occlusion regions due to glasses are considerable in the input facial 
images T , the reconstruction images f will include many errors thereon so that 
these may appear to be unnatural and different from the input facial images T . 

3 0 As described above, since problems due to glasses in the input facial images 

T are merely regarded as the matter of glasses frame, many limitations are arisen in 
the conventional methods so that obtaining high quality glassless facial images is 
very difficult. 
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DISCLOSURE OF THE INVENTION 

It is, therefore, an objective of the present invention to provide an image 
processing method using recursive PCA reconstruction, which is capable of obtaining 
glassless color facial images with a high quality resolution similar to input glasses 
5 color facial images, by removing all occlusion regions including not only glasses 
frame region but also occlusion regions due to reflection by lens and shades by 
glasses within the input glasses color facial images. 

In accordance with the present invention, an image processing method for 
obtaining a glassless image from a color frontal facial image bearing glasses, 

1 0 comprising the steps of: a) receiving an RGB color frontal facial image bearing 

glasses, wherein RGB are red-, green-, and blue-component contained in the received 
RGB color frontal facial image; b) extracting candidates of eye regions from the 
received RGB color frontal facial image; c) determining an exact eye region out of 
the candidates and normalizing the received RGB color frontal facial image in a 

1 5 predetermined size by centering on the determined eye region; d) extracting a glasses 
frame region by using color information contained in the received RGB color frontal 
facial image and edge information of a glasses frame; e) performing an RGB-HSI 
transformation on the normalized frontal facial image; f) generating H 1 , S', and I f 
glassless compensated images on the basis of the RGB-HSI transformed H-, S-, and 

2 0 I-component normalized frontal facial images, wherein the H-, S-, and I-component 

represent a hue, a saturation, and an intensity, .respectively; g) obtaining R f , G', and B f 
compensated images by performing an HSI-RGB transformation on the H', S f , and I 1 
glassless compensated images; and h) creating a glassless final color facial image on 
the basis of the R f , G f , and B r compensated images. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other objects and features of the present invention will 
become apparent from the following description of preferred embodiments given in 
conjunction with the accompanying drawings. 

3 0 Fig. 1 is input facial images with glasses and glassless reconstruction facial 

images obtained according to a conventional image processing method using PCA 
reconstruction. 

Fig. 2 is a flow diagram of a process of removing glasses from color input 
facial images in accordance with an image processing method, which employs 
3 5 recursive PCA reconstruction of the present invention. 

Fig. 3 is a diagram for explaining the recursive PCA reconstruction of 
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processing normalized facial images of an intensity component (I-component) in 
accordance with the present invention. 

Fig. 4 is a diagram of a process of extracting a glasses frame region from 
each of the color input facial images in accordance with the present invention. 
5 Fig. 5 is a criterion for determining an occlusion region in each differential 

image on the basis of a gray-level in accordance with the present invention. 

Figs. 6a to 6c are images classified into skin and non-skin color regions by 
using color information of the color input facial images in accordance with the 
present invention. 

1 0 Fig. 7 is a range of weights for compensating the reconstruction facial 

images in accordance with the present invention. 

Fig. 8 is a flow diagram for processing normalized facial images of 

saturation and hue components (S- and H-components) in the recursive PCA 

reconstruction in accordance with the present invention. 
15 Fig. 9 is the I-component normalized images and I-component compensated 

images obtained from the I-component input facial images in accordance with the 

present invention. 

Fig. 10 is the input color facial images and final color images obtained from 
the input facial images in accordance with the present invention. 
2 0 Fig. 1 1 is a block diagram of an image processing system in accordance 

with the present invention. 

Fig. 12 is S-component normalized images and S-component compensated 
images obtained from the S-component normalized images in accordance with the 
present invention. 

2 5 Fig. 13 is H x -vector-component normalized images and H x -vector- 

component compensated images obtained from the H x - vector-component normalized 
images in accordance with the present invention. 

Fig. 14 is Hy-vector-component normalized images and H y -vector- 
component compensated images obtained from the H y - vector-component normalized 

3 0 images in accordance with the present invention. 

BEST MODE FOR CARRYING OUT THE INVENTION 

Now, embodiments in accordance with the image processing method of the 
present invention will be described in detail with reference to Figs. 2 to 14. 
3 5 Referring first to Fig. 11, which is a block diagram of an image processing 

system employing recursive principal component analysis (PCA) reconstruction in 
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accordance with the present invention. As shown, image processing system 1000 in 
accordance with the present invention comprises image input unit 1002, image 
processing unit 1004, image output unit 1006, and face database (DB) 2000. 

Image input unit 1002 receives color input facial images with glasses and 
5 transmits them to image processing unit 1004. Image input unit 1002 is 

implemented with a conventional image input device such as a digital camera. 
Image processing unit 1004 performs an image process in accordance with the 
present invention on the color input facial images received from image input unit 
1002 and generates glassless color facial images. Image processing unit 1004 is 

1 0 implemented with a conventional computing device. Face DB 2000 stores the color 
input facial images and intermediate images of the color input facial images 
generated during the image process performed by image processing unit 1 004. Face 
DB 2000 also stores glassless sample facial images, which are used in the recursive 
PC A reconstruction to be described in below. Image output unit 1006 outputs 

15 glassless color facial images, which are generated by image processing unit 1004. 

Image output unit 1006 is implemented with a conventional display device such as a 
monitor. 

Referring to Fig. 2, which is a flow diagram of a process of removing 
glasses from input color facial images in accordance with an image processing 
2 0 method of the present invention. The image processing method employs recursive 
PCA reconstruction in accordance with the present invention. Image processing 
system 1000 first receives color input facial images with glasses from image input 
unit 1002 (step S202). 

Image processing unit 1004 of image process system 1000 obtains binary 

2 5 generalized skin color distribution (GSCD) transform images, which enhance skin 

colors of faces in the color input facial images by using color information contained 
therein (step S204). Also, image processing unit 1004 obtains binary black and 
white color distribution (BWCD) transform images, which enhance black and white 
colors of faces in the color input facial images by using the color information (step 

3 0 S206). Obtaining the binary GSCD and BWCD transform images from the color 

input facial images is performed by a known method in the art. 

In order to find candidates of eye regions required for normalizing the color 
input facial images, image processing unit 1004 removes certain regions having 
colors different from the skin color, such as eye, eyebrow, mouth, or slipping-down 
3 5 hairs, from the binary GSCD transform images by performing a morphology filtering 
(step S208). Image processing unit 1004 extracts the candidates from the color 
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input facial images by using the binary B WCD transform images and the 
morphology filtered GSCD transform images (step S210). Image processing unit 
1004 determines exact eye regions out of the candidates and normalizes the color 
input facial images with a predetermined size by centering on the exact eye regions 
(step S214). The normalized color input facial images are represented as primary 
color components of red, green, and blue in a red-green-blue (RGB) model. 

Image processing unit 1004 generates normalized hue (H), saturation (S), 
and intensity (I) components input facial images of by performing an RGB-HSI 
transformation on the normalized color input facial images (step S216). As is well 
known in the art, RGB model images are easily transformed into HSI model images 
and vice versa. In the present invention, in order to process the color input facial 
images, the RGB-HSI transformation is performed on the normalized color input 
facial images by using Equation 6. 



15 



7 = i(i? + G + i?) 
5 = 1-- 



(R + G + B) 
H = cos -1 ' 



[min(R + G + B)] 



±[(R-G)+(R-B)] 



[(R-G) 2 +(R-B)(G-B)Y 2 



(Eq.6) 



if,{B>G) 
then 

H = 360°-H 



wherein H-component has a value of Oto 360 degrees, S-component has a value of 0 
to 1, and I-component has a value of 0 to 255. 

Meanwhile, image processing unit 1004 extracts glasses frames using the 
2 0 color information of the color input facial images and edge information of the glasses 
frames in the color input facial images (step S2 1 2). Detailed description of this 
procedure will be made with reference to Fig. 4. 

Referring to Fig. 4, image processing unit 1004 performs an AND operation 
on the morphology filtered GSCD transform images obtained at step S208 and the 
2 5 binary BWCD transform images obtained at step S206, to thereby obtain image 400. 
Image 400 shows black and white regions in the color input facial images and 
includes eye and eyebrow regions. Next, image processing unit 1 004 performs an 
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OR operation on image 400 and image 100 that is a GSCD transform image of gray- 
level obtained at step S204, to thereby obtain image 801. Image 801 is an image of 
which eliminates eyes and eyebrows from image 100. In order to accurately 
represent the glasses frames included in the color input facial images, image 
5 processing unit 1004 detects edges of the glasses frame in image 801 through the 
well-known Sobel method to generate image 802. Image processing unit 1004 
performs an OR operation on image 802 and an inversion image of image 801 to 
obtain image 803, and then normalizes image 803 with the same size as the 
normalized color input facial images at step S214, to thereby obtain image 800 

1 0 (hereinafter, referred to glasses frame image G 800) using location information of the 
glasses frame. Glasses frame image G 800 contains a glasses frame region only. 

Referring back to Fig. 2, image processing unit 1004 obtains compensated 
images of H-, S-, and I-components by applying the recursive PCA reconstruction 
with respect to the normalized color input facial images of H-, S-, and I-components 

1 5 (step S2 1 8). Image processing unit 1 004 then performs the HSI-RGB 

transformation on the compensated image of H-, S-, and I-components to obtain final 
glassless color facial images (step S220). 

Referring to Fig. 3, which shows a diagram for explaining the recursive 
PCA reconstruction in which image processing unit 1004 processes the normalized I- 

2 0 component input facial images out of the RGB-HSI transform images at step S218. 
First, image processing unit 1004 reconstructs the normalized I-component input 
facial images according to the conventional PCA algorithm. Hereinafter, the 
normalized I-component input facial images are referred to I-component normalized 
images T(i) 500, wherein i is an index to indicate pixels within I-component 

2 5 normalized images T(i) 500. That is, resultant images reconstructed according to 

Equation 5 becomes images corresponding to 601 of Fig. 3 (hereinafter, referred to I- 

component reconstructed images f(z) 601). Using Equation 7, image processing 
unit 1004 calculates differential images between I-component normalized images 
r(i) 500 and I-component reconstructed images f(z) 601, to obtain images 

3 0 corresponding to 602 of Fig. 3 (hereinafter, referred to I-component differential 

images d(i) 602). 



rf(0 = |r(o-f(o| 
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(Eq. 7). 
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As shown, glasses frames overlapping with eyebrows are not extracted 
completely in I-component differential images d(i) 602. This results from a 
lower difference in gray-level between I-component normalized images 500 

and I-component reconstructed images t (i) 601 since the glasses frames 

overlapping with the eyebrows in I-component normalized images r(z) 500 are 
represented as eyebrows having low gray-level in I-component reconstructed images 

r(z) 60 1. If the glasses frames overlapping with the eyebrows are not removed 

completely, facial features may be degraded so that obtaining reconstructed images 
similar to I-component normalized images T(z) 500 is very difficult. In order to 
accurately find regions of the glasses frames overlapping with the eyebrows, i.e., 
occlusion regions, the present invention employs glasses frame image G 800 (shown 
in Fig. 4) extracted at step S212 of Fig. 2. 

Image processing unit 1004 stretches I-component differential images d(i) 
602 after reflecting gray-level information corresponding to the face therein, to 
thereby create images corresponding to 603 of Fig. 3 (hereinafter, referred to I- 
component differential images D(i) 603) according to Equation 8. That is, I- 
component differential images D(i) 603 are created by taking square roots to the 
product of gray-level intensities with respect to I-component differential images 

d(i) 602 and I-component reconstructed images t(i) 601. 



Using I-component differential images D(i) 603 is advantageous as 
follows. Emphasizing a difference of gray-level intensities with respect to the 
occlusion regions due to the glasses frames is possible and thus the occlusion regions 
are easily removed. This results in obtaining more natural looking glassless images. 
Further, decreasing variation of gray-level intensities with respect to the facial 
features, such as eyes and eyebrows, in I-component normalized images T(i) 500 is 
possible so that I-component normalized images T(i) 500 can be used as these 
stand. 

In order to include glasses frame image G 800 in I-component differential 
images D(i) 603, classifying I-component differential images D(i) 603 into 
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occlusion regions and non-occlusion regions is necessary. Error distribution in I- 
component differential images D(i) 603 appears in the occlusion regions due to the 
glasses greater than the non-occlusion regions. With error distribution, the 
occlusion and non-occlusion regions are classified in a gray-level range of 0 to 255 
5 shown in Fig. 5. Thresholds for classifying the regions are determined by Equation 
9. 

T L =mean(DUy), 
where, j e skin region 

T H =mean(D(k)) 9 (Eq * 9 ) 

where, k e {j\D(j) > T L }, j e non - skin region 

10 wherein T L and T H are lower and upper thresholds, respectively; D(J) is error 
values in the non-occlusion regions; and D(k) is error values in the occlusion 
regions. 

Detailed description of calculating the lower and upper thresholds T L and 
T H will be followed. In order to find non-occlusion regions in I-component 
1 5 normalized images T(i) 500, image processing unit 1 004 performs OR operation 
with an inversion image (shown in image (a) of Fig. 6) of the binary GSCD 
transform image, and the binary BWCD transform image (shown in image (b) of Fig. 
6). Once more, image processing unit 1004 normalizes the resultant image with the 
same size of I-component normalized images 500, centering on locations of 

2 0 eyes, to obtain an image (shown in image (c) of Fig. 6). In image (c) of Fig. 6, dark 

regions are obtained by emphasizing the skin color in I-component normalized 
images T(i) 500 and belong to the non-occlusion regions. Accordingly, the lower 
threshold T L is obtained by computing an average of errors occurred in the non- 
occlusion regions within I-component differential images Z>(z) 603. 
25 On the other hand, the occlusion regions by the glasses are obtained by 

emphasizing the non-skin colors in I-component normalized images T{i) 500 and 
expressed as white regions in image (c) of Fig. 6. Accordingly, the upper threshold 
T H is obtained by computing an average of errors that are greater than the lower 
threshold T L within I-component differential images D(i) 603. 

3 0 Here, uncertain regions still exist in I-component differential images D(i) 

603. The uncertain regions contain errors greater than the lower threshold T L and 
smaller than the upper threshold T H . The glasses frame regions are not contained 
in the occlusion regions since values of gray-level of the glasses frame regions in I- 
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20 



component differential images D(i) 603 are likely to include errors less than the 
upper threshold T H . Therefore, in order to include the glasses frame regions in the 
vicinity of eyebrows into the occlusion regions, glasses frame image G 800 extracted 
at step S212 is used. In Equation 10, where values out of errors in I-component 
differential images D{i) 603 have a gray-level less than the upper threshold T H ,a. 
value G(i) having a gray-level greater than the upper threshold T H is used in 
glasses frame image G 800. Images obtained through the above described 
procedures are I-component differential images D'(i) 604 shown in Fig. 3. 

K>W)<T H 
then 

Z)'(0 = max(Z)(0,G(0), 

where, i = \,--,N (Eq. 10) 

if, D(i)*T H 

then 

£>'(0 = £>(/) 



I-component differential images D'(i) 604 are used for removing the 
occlusion regions due to the glasses I-component normalized images 500. 
Error values in I-component differential images D'(i) 604 have a gray-level range 
from 0 to 255. The error values are classified into non-occlusion regions, occlusion 
15 regions, and uncertain regions on the basis of the defined thresholds T, and T 

And then, according to Equation 1 1, a different weight is given to the non-occlusion, 
occlusion, and uncertain regions of I-component differential images D'(i) 604. 

If, D'(i)>T H 
then 
o(0 = l 

if,T L <,D'(i)<T H 

(Eq. 11) 

^(0 = 1-0.5^^(0 
else 

(o{i) = 0 

wherein co(j) are weights with respect to I-component differential images D'(i) 
604. 
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Weights of 1 are given to the occlusion regions having error values greater 
than the upper threshold T H and weights of 0 are given to the non-occlusion regions 
having error values smaller than the lower threshold T L . Herein, the weights of 0 
indicate that no change occurs in the original input facial images. Weights of 0:5 to 
5 1 are given to the uncertain regions. Although the lowest weights, i.e., 0.5, was 
determined by experiments, but not limit thereto, employing values capable of 
compensating unnatural looking facial images after removing the occlusions due to 
the glasses within I-component normalized images T(i) 500 is possible. This 
compensates only regions considered as parts of the glasses within I-component 
10 normalized images T(i) 500. As described above, the weights (shown in Fig. 7) 
given by Equation 1 1 are used for compensating the occlusion regions due to the 
glasses within I-component differential images D'(i) 604 according to Equation 12. 

If (r = 0) 
then 

r/(r) = co • <p + (1 - a>) • r(i) (Eq. 12) 

else 

r/(0 = o> f,(o+(i-®) r(o 

15 

wherein r,'(0 is I-component compensated images 605 without bearing the glasses 

in accordance with the present invention. 

Where weights are 0, regions are determined as the non-occlusion regions 
so that I-component normalized images T(0 500 are used. Where weights are 1, 
2 0 regions are determined as the occlusion regions. In the case of the occlusion 

regions, at a first iteration (t=0) (i.e., a first column of Fig. 3) of the recursive PCA 
reconstruction in accordance with the present invention, I-component compensated 
images 605 are obtained using I-component average image <p 700, which is 
calculated from the sample facial images T N . The reason why I-component 

2 5 average image <p 700 is used is that I-component reconstructed images f(z) 601 

are not glassless images in which the glasses included in I-component normalized 
images T(i) 500 are completely removed. Accordingly, from the second iteration 
(i.e., the second column shown in Fig. 3), the glasses are removed using I-component 

reconstructed images f (i) 601 , which are obtained by reconstructing I-component 
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compensated images 605 obtained at the first iteration (t=0). 

Where weights have a range from 0.5 to 1, regions are determined as the 
uncertain regions. Thus, the glasses are removed such that the weights are applied 
to the combination of I-component normalized images r(z) 500 and I-component 
5 average image <p 700, or I-component normalized images T(i) 500 and I- 

component reconstructed images f(z) 601. Specifically, at the first iteration (t=0) 

of the inventive recursive PCA reconstruction, the glasses are removed by using the 
sum of resultant values upon multiplying a weight a to gray-level intensities of the 
uncertain region within I-component average image <p 700, and upon multiplying a 
1 0 weight (1 - co ) to gray-level intensities of regions within I-component normalized 
image T{i) 500, the regions corresponding to the uncertain region within I- 
component average image <p 700. From the second iteration, the recursive PCA 
reconstruction is performed by using I-component normalized image T(i) 500 and 

I-component reconstructed image f(z) 601 reconstructed from I-component 

1 5 compensated image 605 at the first iteration (t=0). 

For example, in the first column of Fig. 3 except I-component normalized 

image r(z') 500, I-component reconstructed and compensated images f (z) 601 to 

605 are generated such that finding the glasses regions within I-component 
normalized image T(i) 500 is performed and then processing the glasses regions is 
20 performed using I-component average image <p 700 as described above. I- 

component reconstructed and compensated images f (z) 601 to 605 in the second 

column of Fig. 3 are generated on the basis of I-component reconstructed image 

f (0 601 reconstructed from I-component compensated image 605 at the first 

iteration (t=0) and I-component normalized images T(i) 500. Such procedure is 
2 5 applied to subsequent columns from the third iteration. The inventive recursive 

PCA reconstruction is repeated until a difference between I-component reconstructed 

images f(i) 601 is less than a predetermined value 9 or it is constantly 
maintained as defined in Equation 13. 
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r,(0-r/ + i(o \\<0 



(Eq. 13) 



When the iteration of the recursive PCA reconstruction stops, compensated 
facial image, which is generated at the last iteration shown at right-bottom of Fig. 3, 
5 i.e., one of I-component compensated images 605, becomes an I-component final 
compensated image. Examples of I-component final compensated images are 
shown in Fig. 9. Hereinafter, the I-component final compensated image will be 
referred as an I 1 image. 

Referring to Fig. 8, which is a flow diagram for processing S- and H- 

1 0 components normalized images obtained at step S216 in accordance with the 

inventive recursive PCA reconstruction. In order to obtain glassless color images, it 
is necessary to perform the recursive PCA reconstruction with respect to not only the 
I-component normalized images as described with reference to Fig. 3, but also the S- 
and H-components normalized images. Unlike the afore-mentioned recursive PCA 

1 5 reconstruction with respect to the I-component normalized images, glasses frame 

image G 800 extracted at step S212 shown in Fig. 2 is not used in the recursive PCA 
reconstruction with respect to the S- and H-components normalized images. The 
reason why glasses frame image G 800 is not used that occlusion regions obtained 
from the S- and H-components images are different from those of the I-component 

2 0 normalized images. 

First, detailed description of the recursive PCA reconstruction with respect 
to the S-component normalized image will be followed. Hereinafter, the S- 
component normalized image shown in Fig. 12 will be referred as an S-component 
normalized image T(i) . Before performing the recursive PCA reconstruction, 

2 5 image values of the S-component normalized image r(i) containing glasses are 

stretched to have values from 0 to 255. 

At step S802, image processing unit 1004 shown in Fig. 11 reconstructs the 
S-component normalized image , which is obtained at step S216 of Fig. 2, by 
using an S-component average image <p and sigenfaces u k through Equation 5, to 

3 0 thereby create an S-component glassless reconstructed image f (/) . The S- 

component average image q> and eigenfaces u k are calculated on the basis of the 
sample facial images r„ stored on face DB 2000 shown in Fig. 1 1 . 

At step S804, image processing unit 1004 obtains an S-component 
differential image d(i) from the S-component normalized image T(z) and the S- 
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component reconstructed image f (i) generated at step S802, by using Equation 7. 

At step S806, the image processing unit 1004 stretches the S-component differential 
image d(i) by using Equation 8, to thereby obtain an S-component differential 
image D(i) in which facial features are reflected. 
5 As step S808, image processing unit 1 004 calculates an average of errors 

within non-occlusion regions of the S-component differential image D(i) by using 
Equation 9, to thereby obtain the lower threshold T L . Image processing unit 1004 
further computes an average of errors greater than the lower threshold T L within the 
S-component differential image D(i) , to thereby obtain the upper threshold T H , at 
10 stepS808. 

At step S810, image processing unit 1004 applies a weight of 1 on occlusion 
regions of the S-component differential image D(i) , a weight of 0 on the non- 
occlusion regions, and a weight having a value of 0.5 to 1 on uncertain regions by 
using Equation 10, to thereby generate an S-component compensated image. 
1 5 Similar to the processing of the I-component normalized image, an image 

reconstructed from the S-component compensated image, which is generated at the 

first iteration (t=0), is used as an S-component reconstructed image f(0 at the 

second iteration of the recursive PC A reconstruction. 

At step S812, image processing unit 1004 determines, using Equation 13, 
2 0 whether a difference between the S-component reconstructed images generated by 

the recursive PCA reconstruction is less than or equal to a predetermined value 0 . 

Where the difference is less than the predetermined value 6 , image processing unit 

1004 stops the recursive PCA reconstruction to obtain the current compensated 

image as an S-component final compensated image (hereinafter, referred as an S f 
2 5 image). The S' image is shown in Fig. 12. Otherwise, image processing unit 

1004 returns to step S802. 

An H-component normalized image is processed similar to the S-component 

normalized image described with reference to Fig. 8. Herein, H-component is 

represented as vectors of (H x ,H y ) T using Equation 14. 

30 

H x =cos(H) 

H y =sin(H) C^ 14 ) 
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As is well known in the art, in HIS model, H-component is expressed in an 
angular coordinate system in which a zero and 360 degrees show the same color 
different from I- and S-components. In case of applying the recursive PCA 
reconstruction to the H-component normalized image, it is necessary to do not 
5 discriminate a color at zero degree from a color at 360 degrees. Therefore, the 

present invention utilizes the vectors of (H x9 H y f instead of the H-component 
H x - and H y -vector-component normalized images are shown in Figs. 13 and 14, 

respectively. Similar to the S-component normalized image, the i^-and H y - 
vector-component normalized images are stretched to have values of 0 to 255, before 
1 0 applying the recursive PCA reconstruction thereto. H' x - and H y -vector- 
component compensated images obtained by processing the H - and H -vector- 

x y 

component normalized images according to the recursive PCA reconstruction are 
shown in Figs. 13 and 14, respectively. 

In order to obtain an H-component final compensated image (hereinafter, 

15 referred as an H' image), restoring H' x -and H y -vector-component compensated 

images which were stretched is performed to apply the recursive PCA reconstruction 
thereto. Using Equation 15, the H' image is obtained with respect to the H- 
component normalized image. 



20 



H' = cos-\H" x ) 



(Eq. 15). 



Herein, since the H' x - and H' y -vector-component compensated images do not 

satisfy the relationship of H' x 2 + H' y 2 , H' x is normalized to (h' x 2 +H' y 2 f*, thereby 

obtaining H" . Then, H' is calculated within a range of 0 to 360 degrees on the 
25 basis of H" x . 

Using Equation 16, the HSI-RGB transformation is performed on the if' , 
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iS",and /' images. The S' and /' images are subjected to serve the restoring 
process similar to the H' image. After the HSI-RGB transformation, R' -, G'-, 
and B' -component images are obtained. Using Equation 1 6, a final color image is 
obtained on the basis of the R' -, G' -, and B' -component images. This process is 
5 widely known in the art so that detailed explanation will be omitted herein. 



If,0</T<120° 
then 

b = -(l-S') 

r » |(1 + [(6" cos H')/ (cos(60° -H'))]) 

g = l-(b+r) 

if,120 o <#'<240° 

then 

H' = H' -120° 

b = l-(r+g) 

if, 240° <H'<> 360° 

then 

#' = #'-120° 

b =|(1 +[(5'cosfl r ')/(cos(60°-/f'))]) 

r-l-(g + *) ^ 16 > 

where 

R' = 3I'r 

G' = 3I'g 

B' = 3I'b 



wherein r, g, and b are normalization values within a range of 0 to 1, which satisfy 
1 0 r+g+b=l , respectively. Equations 6 and 16 are well known in the art and detailed 
explanation will be omitted herein. (See R. C. Gonzalez and R. E. Woods, "Digital 
Image Processing," Addison- Wesley Publishing Company, 1992). 
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INDUSTRIAL APPLICABILITY 

As described above, in accordance with the present invention, natural 
looking high quality glassless color frontal facial images are obtained by 
simultaneously finding occlusion regions to be removed within input facial images 
5 and compensating the same. The occlusion regions include not only a glasses frame 
region but also reflection and shading regions due to lens and glasses. Further, the 
image processing method in accordance with the present invention is used to resolve 
other various occlusion problems and improves the recognition efficiency of 
automatic face recognition systems. 
1 0 While the present invention has been shown and described with respect to 

the particular embodiments, those skilled in the art will recognize that many changes 
and modifications may be made without departing from the scope of the invention as 
defined in the appended claims. 



-18- 



WO 2004/040502 



PCT/KR2003/000927 



CLAIMS 



1 . An image processing method for obtaining a glassless image from a color 
frontal facial image bearing glasses, comprising the steps of: 
5 a) receiving an RGB color frontal facial image bearing glasses, wherein 

RGB are red-, green-, and blue-component contained in the received RGB color 
frontal facial image; 

b) extracting candidates of eye regions from the received RGB color frontal 
facial image; 

10 c) determining an exact eye region out of the candidates and normalizing the 

received RGB color frontal facial image in a predetermined size by centering on the 

determined eye region; 

d) extracting a glasses frame region by using color information contained in 

the received RGB color frontal facial image and edge information of a glasses frame; 
15 e) performing an RGB-HSI transformation on the normalized frontal facial 

image; 

f) generating H f , S ! , and T glassless compensated images on the basis of the 
RGB-HSI transformed H-, S-, and I-component normalized frontal facial images, 
wherein the H-, S-, and I-component represent a hue, a saturation, and an intensity, 

2 0 respectively; 

g) obtaining R, G\ and B' compensated images by performing an HSI-RGB 
transformation on the H ! , S f , and T glassless compensated images; and 

h) creating a glassless final color facial image on the basis of the R, G f , and 
B 1 compensated images, 

2 5 wherein the step f) further comprises the steps of: 

fl) obtaining S-, and I-component reconstructed images by 
reconstructing the H-, S-, and I-component normalized frontal facial images; 

£2) obtaining H-, S-, and I-component first differential images between the 
H-, S-, and I-component normalized frontal facial images and the H-, S-, I- 

3 0 component reconstructed images; 

f3) obtaining H-, S-, and I-component second differential images by 
stretching H-, S-, and I-component first differential images on the basis of pixel 
information contained in the H-, S-, and I-component first differential images; 

f4) determining thresholds to classify the H-, S-, and I-component second 
3 5 differential images into occlusion regions, non-occlusion regions, and uncertain 
regions; 
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10 



f5) obtaining an I-component third differential image by including the 
extracted glasses frame region onto the uncertain region classified by the threshold 
within the I-component second differential image; 

f6) classifying the H- and S-component second differential images and the I- 
component third differential image on the basis of the thresholds to determine a 
weight to be applied on the respective- classified images; and 

f7) obtaining the H f , S f , and V compensated images by applying the weight 
on each of the H- and S-component second differential images and the I-component 
third differential image. 

2. The image processing method of Claim 1, wherein the H-component 
normalized frontal facial image is expressed as H x - and H y -vector-component 
images as follows: 

H x =cos{H) 
H y =sin(#) 

wherein H' x and H' y compensated images are obtained on the basis of the H x - 
and H y -vector-component images. 



20 3 • The image processing method of Claim 2, wherein the HP-component 
glassless compensated image is obtained from the H' x and H y compensated 
images as follows: 

TJf 

H" = 
X JH>1 + H? 



15 



25 



wherein H" x is obtained by normalizing H' x with (H'j + H?)** . 



4. 



The image processing method of Claim 1, wherein the IT, S', and I' 
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compensated images are obtained by a principal component analysis (PCA) 
reconstruction, and 

the step c) further comprises the steps of: 

cl) obtaining first and second transformed images on the basis of color 
information contained in the received RGB color frontal facial image; and 

c2) normalizing the received RGB color frontal facial image by using the 
first and second transformed images. 

5. The image processing method of Claim 4, wherein the first and second 
transformed images are a generalized skin color distribution (GSCD) image and a 
black and white color distribution (BWCD) image, respectively. 

6. The image processing method of Claim 5, wherein the color information is 
gray-level pixel information. 

7. The image processing method of Claim 5, wherein the stretching at step f3) 
is performed as follows: 

D(i) = {f(i)d(i)f 

2 0 wherein D(i) represents the H-, S-, and I-component second differential images, 

represents the H-, S-, and I-component reconstructed images generated during 

performing the PCA reconstruction, d(i) represents the H-, S-, and I-component first 
differential images, and i is an index for indicating pixels in each image. 

25 ,8. The image processing method of Claim 5, wherein the occlusion regions 

contain the glasses and errors due to a reflection and shade by the glasses within the 
received RGB color frontal facial image. 

9. The image processing method of Claim 8, wherein the step f4) further 

3 0 comprises the steps of: 

inversing the first transformed image; 

performing an OR operation on the inversed first transformed image with 
the second transformed image; 

determining an average of errors within the OR operated image as a lower 
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threshold; and 

determining an average of errors greater than the lower threshold within the 
OR operated image as an upper threshold. 

5 10. The image processing method of Claim 9, wherein the upper and lower 
thresholds are determined as follows: 

T L =mean(D(j)) 9 
where, j e skin region 
T H = mean(D(k)) 9 

where, k e {j\D(J) > T L \ j g non - skin region 

wherein T L and T H represent the lower and upper thresholds, respectively, D(j) 
represents errors of skin regions corresponding to the non-occlusion regions within 
the S-, and I-component second differential images, and D{k) represents errors 
of non-skin regions corresponding to the occlusion regions within the H-, S-, and I- 
component second differential images. 

1 1 . The image processing method of Claim 10, wherein the occlusion regions in 
the I-component second differential image includes the glasses frame region, which 
is extracted as follows: 

lf,D(i)<T ff 
then 

D'(i) = max(D(i),G(i)) 9 
2 0 where,/ = 1,—,JV 

if, D(i)>T ff 
then 

D'(i)=D(i) 

wherein the I-component third differential image is obtained on the basis of the I- 
component second differential image including the glasses frame region, D f (i) 
represents the I-component third differential image, and G(i) represents gray-level 
2 5 values of the extracted glasses frame region. 



10 



15 



The image processing method of Claim 11, wherein the weights are 
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determined as follows: 

If, D'(i)>T H 
then 

«(0=i 

if,T L ZD'(i)<T„ 
then 

^(0 = 1-0.5^-^ 
T„~T L 

else 

a>{i) = 0 



5 wherein a weight of 1 is applied on regions having error values greater than the upper 
threshold within the I-component third differential image, a weight of 0 on regions 
having error values less than the lower threshold, and a weight with a value from 0.5 
to 1 on regions having error values between the upper and lower thresholds, and 

wherein the regions having the error values greater than the upper threshold 
10 are the occlusion regions, the regions having the error values less than the lower 
threshold are the non-occlusion regions, and the regions having the error values 
between the upper and lower thresholds are the uncertain regions. 

13. The image processing method of Claim 12, wherein the occlusion regions 
1 5 within the I-component third differential image are compensated as follows: 

If (* = 0) 
then 

r;(o = ^+(i-a>)r(o 

else 

r/(o=*> fxo+(i-*>)-r(o 

wherein T' t (i) represents the T compensated image, co represents the weights, <p 
2 0 represents an I-component average image, f t (i) is the I-component reconstructed 
image, and T(i) is the I-component normalized frontal facial image. 

14. The image processing method of Claim 10, wherein a weight of 1 is applied 
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on regions having error values greater than the upper threshold within the H- and S- 
component second differential images, a weight of 0 on regions having error values 
less than the lower threshold, and a weight with a value from 0.5 to 1 on regions 
having error values between the lower and upper threshold, 
5 wherein the regions having the error values greater than the upper threshold 

are the occlusion regions, the regions having the error values less than the lower 
threshold are the non-occlusion regions, and the regions having the error values 
between the upper and lower thresholds are the uncertain regions. 

10 15. The image processing method of Claim 1 4, wherein the occlusion regions 
within the H- and S-component second differential images are compensated as 
follows: 

lf(* = 0) 
then 

r/(0=*> -<P+(i-*>)-r(o 

else 

r/(0=a> f t (o+(i-o))-r(o 



15 



20 



wherein T' t {i) represents the H' and S 1 compensated images, a? represents the 
weights, <p represents H- and S-component average images, f f (z) is the H- and S- 

component reconstructed images, and T{i) is the H- and S-component nonnalized 
frontal facial images. 



16. The image processing method of Claim 13 or 15, further comprising the step 
of repeating the steps fl) to f7), 

wherein the IT, S', and T compensated images obtained at step f7) are used 
at step fl) instead of the H-, S-, and I-component normalized frontal facial images 
25 when performing the repeating step. 

17. The image processing method of Claim 16, further comprising the steps of: 
determining whether each difference between the H f , S f , and T compensated 

images obtained at step f7) and HP, S', and F compensated images obtained after 
3 0 performing the repeating step is less than the predetermined value or not; 

performing, if each difference is less than the predetermined value, the HSI- 
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RGB transformation on the H f , S\ and I f compensated images obtained after 
performing the repeating step, to thereby obtain the glassies? final color facial image 
based on the transformed H\ S', and T compensated images obtained after 
performing the repeating step; and 

performing the repeating step, if each difference is greater than the 
predetermined value. 
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